Fix purify stopping before the tensor is pure with a random order - #708

Fixes #702

purify visits the surfaces in a new random order on every sweep and stopped at the first sweep whose total impurity did not shrink compared to the previous sweep. With a random order that total is not monotone, and empty cells make convergence slow, so tensors with empty cells were often left impure, silently. With is_randomized=False the same inputs purify to about 1e-16.

Change

When the order is random, a sweep that fails to shrink the total now ends the loop only once the total has dropped to floating point noise, defined as 1e-12 of the first sweep. Otherwise the loop continues until 20 sweeps in a row fail to set a new minimum, which also guarantees that it terminates.

• The fixed-order path (is_randomized=False) keeps the old exit condition and is unchanged.
• The multiclass loop (PurifyNormalizedMulticlass) had the same exit condition and gets the same change. I confirmed the multiclass case fails on the released library too.
• The logic is in one helper, IsPurifySweepingDone, used by both loops.

I picked the exit rule by simulating the sweep loop in Python on 5x6, 3x3x3, 4x5x6, 8x8x8, 32x32 and 10x10 tensors:

• The current rule fails 91 to 96 of 100 on the tensors with empty cells.
• Switching to a fixed order after the first stall still failed 38 to 61 of 100.
• A patience-only rule (20 sweeps without a new minimum) fixes everything but runs about 2.4 to 2.7x more sweeps on inputs that already worked.
• The chosen rule fixes all of them and takes exactly the same number of sweeps as today on inputs that already worked.

Evidence

Before and after on 100 random tensors each, tolerance=0, is_randomized=True, worst weighted row or column sum:

The "before" numbers are from the released 0.7.8 wheel. The "after" numbers are from a build of this branch.

Tests added:

• PurifyTest.cpp: two cases (single class and multiclass) with 50 pseudo random 5x6 tensors each, 60% empty cells, zero tolerance. They check the weighted row and column sums directly because MeasureImpurity returns NaN for a row or column with no weight. Both fail on the original Purify.cpp (50 and 126 failed checks) and pass with this change.
• test_purify.py: two cases (3 tensor shapes, and a multiclass shape) with randomized order and empty cells. Both fail against the released 0.7.8 wheel and pass with this change.

Checks run, Windows 11:

• Full native test suite (shared/libebm/tests, 348 tests): all pass.
• python/interpret-core/tests/utils: 225 passed, 9 skipped. I excluded test_measure_interactions.py, which needs the dash package.
• An end-to-end ExplainableBoostingClassifier fit with purify_result enabled gives the same accuracy and the same purity on the original and the fixed library, in a similar time (5.4 s and 5.2 s).
• clang-format on the changed C++ files and ruff format --check on the changed Python file are clean. The clang-format I had is version 12, not 16.

Limits

• This machine has Visual Studio 2019 Build Tools, and the project targets v143. Its compiler also rejects an unrelated overload in Objective.hpp. I built the library and the tests with clang (via the ziglang package) for x86_64 Windows, CPU compute unit only, with the same flags as build.sh (-std=c++11 -O3) and libebm_exports.def. The AVX2 and AVX-512 units, the MSVC build, Linux, macOS, and the R package were not built or run here.
• The ebm interaction tensors in my end-to-end check were already pure, so that check shows no change for them, not that the fix helps in training. purify_result is off by default, as the issue notes.
• 20 sweeps and 1e-12 are chosen from simulation. I can change either if you prefer other values.